Selecting Weighting Factors in Logarithmic Opinion Pools
نویسنده
چکیده
A simple linear averaging of the outputs of several networks as e.g. in bagging 3], seems to follow naturally from a bias/variance decomposition of the sum-squared error. The sum-squared error of the average model is a quadratic function of the weighting factors assigned to the networks in the ensemble 7], suggesting a quadratic programmingalgorithm for nding the \optimal"weighting factors. If we interpret the output of a network as a probability statement, the sum-squared error corresponds to minus the loglikelihood or the Kullback-Leibler divergence, and linear averaging of the outputs to logarithmic averaging of the probability statements: the logarithmic opinion pool. The crux of this paper is that this whole story about model averaging , bias/variance decompositions, and quadratic programming to nd the optimal weighting factors, is not speciic for the sum-squared error, but applies to the combination of probability statements of any kind in a logarithmic opinion pool, as long as the Kullback-Leibler divergence plays the role of the error measure. As examples we treat model averaging for classiication models under a cross-entropy error measure and models for estimating variances.
منابع مشابه
On Irrelevance of Alternatives and Opinion Pooling
We consider the problem of combining k subjective probability distributions for a quantity of interest θ ∈ Θ. Two modified versions of the irrelevance of alternatives axiom are introduced by focusing on the consensual conditional odds, rather than the consensual probabilities. A strong modification of the axiom requires that the consensual odds depend only on the individual odds, while a weak m...
متن کاملLearning a Product of Experts with Elitist Lasso
Discriminative models such as logistic regression profit from the ability to incorporate arbitrary rich features; however, complex dependencies among overlapping features can often result in weight undertraining. One popular method that attempts to mitigate this problem is logarithmic opinion pools (LOP), which is a specialized form of product of experts model that automatically adjusts the wei...
متن کاملAggregation Under Bias: Rényi Divergence Aggregation and Its Implementation via Machine Learning Markets
Trading in information markets, such as machine learning markets, has been shown to be an effective approach for aggregating the beliefs of different agents. In a machine learning context, aggregation commonly uses forms of linear opinion pools, or logarithmic (log) opinion pools. It is interesting to relate information market aggregation to the machine learning setting. In this paper we introd...
متن کاملMarket Scoring Rules Act As Opinion Pools For Risk-Averse Agents
A market scoring rule (MSR) – a popular tool for designing algorithmic prediction markets – is an incentive-compatible mechanism for the aggregation of probabilistic beliefs from myopic risk-neutral agents. In this paper, we add to a growing body of research aimed at understanding the precise manner in which the price process induced by a MSR incorporates private information from agents who dev...
متن کامل